Operation support system, wearable apparatus, image pickup apparatus, and operation support method

ABSTRACT

To provide an operation support system capable of performing evidence recording of each of performed operations, an operation support system includes a first image pickup apparatus configured to mainly acquire movie data, a second image pickup apparatus configured to mainly acquire still image data, a control apparatus configured to control the first image pickup apparatus and the second image pickup apparatus in cooperation with each other while acquiring operation information about a predetermined operation, and a database storing defined operation information about the predetermined operation in advance, in which the predetermined operation is supported based on image data acquired by causing the first image pickup apparatus and the second image pickup apparatus to cooperate with each other.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of Japanese Application No.2017-157042 filed in Japan on Aug. 16, 2017, the contents of which areincorporated herein by this reference.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an operation support system forsupporting a predetermined operation for which an operation procedure isdefined, a wearable apparatus, an image pickup apparatus, and anoperation support method.

2. Description of the Related Art

Conventionally, various types of industrial products have been inwidespread use. Examples of operations targeting the various types ofindustrial products include various types of operations such asmanufacturing and assembling operations in manufacturing processes andoperations (handling operations) performed when the products are used ormaintenance operations (maintenance, checking, adjustment, repairing,etc.) for the products. For each of the operations, a predeterminedoperation procedure, method, and the like may be clearly defined. On theother hand, the operations include manual operations by an operator.

As such operations, a wide variety of operations exist for each of thetarget products. Each of the operations may require complicatedprocedures or may require a confirmation operation, as needed.Therefore, to reliably perform a defined operation, the operatorentirely and accurately stores a predetermined operation procedure oradvances the operation while referring to a manual such as an operatingmanual.

A strict operation procedure is defined for this type of operation, andthe operation procedure may have to be faithfully implemented.

In the case of a washing operation, for example, a series of operationsincluding performing a washing operation for a predetermined site of atarget instrument and then performing an observation operation forconfirming a washing state of the site is repeated. This type ofoperation has been required to be cleanly, quickly, and smoothlyperformed by reducing an opportunity for contamination.

However, skill is required to reliably remember all of a predeterminedseries of operations and always accurately perform the operations. Whenthe operation is advanced while referring to a manual such as anoperating manual, the operation is prevented from being smoothlyperformed while an operation time period is required.

Various types of operation support systems for supporting an operatorwhen a predetermined operation for which an operation procedure isclearly defined is performed have been proposed by Japanese PatentApplication Laid-Open Publication No. 2017-75001, for example.

The operation support system disclosed by Japanese Patent ApplicationLaid-Open Publication No. 2017-75001 is an operation support system forguiding an operation procedure for a mobile crane. The operation supportsystem is configured using a wearable observation apparatus, forexample, to correctly guide a predetermined operation procedure for anoperator.

The operation support system includes a remote control apparatusincluding a detection device configured to detect a state of anoperating machine, a transmission unit configured to transmit the stateof the operating machine detected by the detection device, an operationunit configured to remotely operate the operating machine, a displayunit configured to display an operation procedure, and a receiving unitconfigured to receive the state of the operating machine transmitted bythe transmission unit, and is configured such that a defined operationprocedure is displayed based on the received state of the operatingmachine on the display unit provided in the remote control apparatus.

Such a configuration enables, for each operation process, detaileddescription of a corresponding operation content to be displayed, asneeded, on a display screen of the observation apparatus in theabove-described operation support system. Thus, even if the operationprocedure has not been memorized in advance, the operation procedure canbe always accurately performed.

However, in the conventional operation support system disclosed byJapanese Patent Application Laid-Open Publication No. 2017-75001,described above, explanatory information prepared in advance, forexample, is merely displayed on the display screen of the observationapparatus.

SUMMARY OF THE INVENTION

The present invention has been made in view of the foregoing and isdirected to providing, in an operation support system for supporting apredetermined operation for which an operation procedure is defined, anoperation support system capable of preforming evidence recording ofeach of performed operations, a wearable apparatus, an image pickupapparatus, and an operation support method.

According to an aspect of the present invention, there is provided anoperation support system, including a first image pickup apparatusconfigured to mainly acquire movie data, a second image pickup apparatusconfigured to mainly acquire still image data, a control apparatusconfigured to control the first image pickup apparatus and the secondimage pickup apparatus in cooperation with each other while acquiringoperation information about a predetermined operation, and a databasestoring defined operation information about the predetermined operationin advance, in which the predetermined operation is supported based onimage data acquired by causing the first image pickup apparatus and thesecond image pickup apparatus to cooperate with each other.

According to an aspect of the present invention, there is provided awearable apparatus, including a first image pickup unit configured toacquire first image data to acquire operation information correspondingto an eye line of an operator, a communication unit configured toacquire second image data from a second image pickup apparatus providedat a different position from a position of the operator, and anoperation judgment unit configured to judge the first image data and thesecond image data according to operation information in a database.

According to an aspect of the present invention, there is provided animage pickup apparatus, including a receiving unit configured to receiveinformation about a first image pickup unit configured to acquire firstimage data to acquire operation information corresponding to an eye lineof an operator by a wearable apparatus, a database storing definedoperation information about a predetermined operation in advance, and anoperation judgment unit configured to judge image data outputted by theimage pickup apparatus and the first image data according to theoperation information in the database.

According to an aspect of the present invention, there is provided anoperation support method, including acquiring first image data toacquire operation information corresponding to an eye line of anoperator, acquiring second image data to acquire operation informationdifferent from the operation information corresponding to an eye line ofthe operator, and judging the first image data and the second image dataaccording to a database storing operation information including imageinformation from a plurality of visual fields relating to apredetermined operation in advance.

According to another aspect of the present invention, there is providedan operation support system, including a wearable terminal apparatusconfigured to acquire movie data in an image pickup range substantiallyequivalent to a field of view of an operator and worn by the operator,an auxiliary camera configured to acquire still image data in apredetermined image pickup range including a predetermined object from adifferent direction from a direction of the wearable terminal apparatus,and a control apparatus configured to control the wearable terminalapparatus and the auxiliary camera in cooperation with each other, inwhich the auxiliary camera is controlled according to the movie dataacquired by the wearable terminal apparatus to acquire still image dataserving as evidence of an operation.

According to another aspect of the present invention, there is providedan operation support method, including acquiring movie data in an imagepickup range substantially equivalent to a field of view of an operator,and acquiring still image data in a predetermined image pickup rangeincluding a predetermined object according to the acquired movie datafrom a different direction from a direction of the movie data asevidence of an operation.

Objectives and advantages of the present invention will become moreapparent from the following detailed description.

According to the present invention, there can be provided, in anoperation support system for supporting a predetermined operation forwhich an operation procedure is defined, an operation support systemcapable of performing evidence recording of each of performedoperations, a wearable apparatus, an image pickup apparatus, and anoperation support method.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic block configuration diagram illustrating aconfiguration of an operation support system according to an embodimentof the present invention;

FIG. 2 is a conceptual diagram illustrating a schematic configuration ofa first image pickup apparatus (a wearable terminal apparatus)constituting a part of the operation support system according to theembodiment of the present invention;

FIG. 3 is a conceptual diagram illustrating a frontal view of anoperator's face with the operator wearing the first image pickupapparatus (the wearable terminal apparatus) and simultaneouslyillustrating a field of view by both the eyes of the operator;

FIG. 4 is a conceptual diagram more specifically illustrating the fieldof view by both the eyes of the operator illustrated in FIG. 3;

FIG. 5 is a flowchart illustrating a flow of operations performed by anoperator in a maintenance operation (an instrument washing andsterilization operation) for a medical instrument;

FIG. 6 is a diagram illustrating an example of evidence informationgenerated by the maintenance operation;

FIG. 7 is a conceptual diagram illustrating how a predeterminedmaintenance operation (the instrument washing and sterilizationoperation) is being performed using the operation support systemaccording to the embodiment of the present invention;

FIG. 8 illustrates a display example of an image based on image dataacquired by the wearable terminal apparatus during the maintenanceoperation (the instrument washing and sterilization operation) using theoperation support system according to the embodiment of the presentinvention;

FIG. 9 is a flowchart illustrating functions of a control apparatus inthe operation support system according to the embodiment of the presentinvention;

FIG. 10 is a flowchart illustrating functions of the wearable terminalapparatus (the first image pickup apparatus) in the operation supportsystem according to the embodiment of the present invention;

FIG. 11 is a flowchart illustrating functions of an auxiliary camera (asecond image pickup apparatus) in the operation support system accordingto the embodiment of the present invention;

FIG. 12 illustrates an example of a database assuming a scissors washingand sterilization operation;

FIG. 13 is a diagram illustrating one scene of a movie acquired by thewearable terminal apparatus when the operator has held a predeterminedtool (a brush), which the operator has in right hand, over a frontsurface (within an image pickup range) of the auxiliary camera toperform a “tool confirmation” operation;

FIG. 14 illustrates a still image acquired by the auxiliary camera at atiming illustrated in FIG. 13;

FIG. 15 illustrates one scene of movie data acquired by the wearableterminal apparatus when a first site (a blade portion) is being brushed;

FIG. 16 is a conceptual diagram illustrating how a predeterminedmaintenance operation (an instrument washing and sterilizationoperation) is being performed using an operation support systemaccording to a modification to the embodiment of the present invention;

FIG. 17 is a diagram illustrating one scene of movie data acquired by awearable terminal apparatus in the modification illustrated in FIG. 16;and

FIG. 18 is a diagram illustrating still image data acquired by anauxiliary camera in the modification illustrated in FIG. 16.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention will be described below according to anillustrated embodiment. The drawings used for the following descriptionschematically illustrate the embodiment. To respectively illustratecomponents with sufficient sizes to be recognizable on the drawings, adimensional relationship, a scale, and the like of each member may bemade to differ for each of the components. Therefore, the presentinvention is not limited only to illustrated forms with respect to thenumber of components described in each of the drawings, a shape of eachof the components, a ratio of the respective sizes of the components, arelative positional relationship among the components, and the like.

First Embodiment

First, a schematic configuration of an operation support systemaccording to an embodiment of the present invention will be describedbelow with reference to FIG. 1. FIG. 1 is a schematic blockconfiguration diagram illustrating a configuration of the operationsupport system according to the embodiment of the present invention.

The operation support system 1 according to the present embodiment is anoperation support system which can provide operation support to reliablyperform a series of predetermined operations including a plurality ofoperations according to a defined procedure and method while performingevidence recording relating to each of the performed operations.

Therefore, the operation support system 1 includes a plurality of imagepickup apparatuses and a control apparatus configured to control theplurality of image pickup apparatuses in cooperation with one another.Note that the present invention provides an example in which theoperation support system 1 includes two image pickup apparatuses (20,30) and one control apparatus (10), as illustrated in FIG. 1.

By the configuration, the operation support system 1 according to thepresent embodiment provides operation support based on a collection ofinformation (a database) about a plurality of operations determined inadvance and a plurality of image data (acquired image data) aboutoperations actually performed. For example, the operation support system1 displays an operation content and an operation procedure of each ofoperations included in a target series of operations to guide theoperation, performs confirmation and judgment for each of theoperations, for example, whether each of the operations has beencorrectly performed and whether a correct operation procedure has beenexecuted, and performs evidence recording when the correct operation hasbeen performed while issuing a warning or the like to an operator whenthe false operation has been performed to force the correct operation.

More specifically, the operation support system 1 according to thepresent embodiment includes the two image pickup apparatuses, i.e., awearable terminal apparatus 20 serving as a first image pickup apparatusand also being a wearable apparatus and an auxiliary camera 30 servingas a second image pickup apparatus and the control apparatus 10configured to control the two image pickup apparatuses in cooperationwith each other. Note that a specific configuration example isillustrated in FIG. 2 for the wearable terminal apparatus 20.

The control apparatus 10 controls each of the two image pickupapparatuses (20, 30), confirms and judges each of the performedoperations based on image data (still image data (second image data),movie data (first image data), etc.) acquired by each of the imagepickup apparatuses (20, 30), and controls an operation such as operationrecording (evidence recording). The control apparatus 10 performscontrol, for example, to operate the two image pickup apparatuses (20,30) in cooperation with each other.

Therefore, the control apparatus 10 includes a control unit 11, acommunication unit 12, a database 13, an evidence recording unit 14, andthe like.

Note that the control apparatus 10 according to the present embodimentincludes various types of components provided in a general controlapparatus, for example, an operation unit and a display unit in additionto the above-described components, illustration of which is omitted.Since the various types of components, illustration of which is omitted,are not directly related to the present invention, detailed descriptionof the components is omitted.

The control unit 11 is a constituent unit including a plurality ofcontrol circuits, for example, configured to integrally control theentire operation support system 1 according to the present embodiment.Examples of the plurality of control circuits included in the controlunit 11 include, particularly, an operation judgment unit 11 a, adisplay control unit 11 b, a guide unit 11 c, an object judgment unit 11d, and a size judgment unit 11 e.

The operation judgment unit 11 a is a constituent unit including acontrol circuit, for example, configured to confirm and judge each ofitems relating to the performed operation based on the image dataacquired by the two image pickup apparatuses (20, 30), informationstored in the database 13 in advance, judgment result informationacquired by the object judgment unit 11 d and the size judgment unit 11e, and the like.

Note that examples of the judgment herein performed include judgment ofthe type of a tool used for each of operations and judgment of the typeof the operation.

The display control unit 11 b is a constituent unit including a controlcircuit, for example, configured to control a display unit (notillustrated) included in the control apparatus 10 or a display unit (25)and a display unit (not illustrated) respectively included in theplurality of (two) image pickup apparatuses (20, 30) cooperating withthe control apparatus 10 to perform predetermined display (warningdisplay, etc. in addition to image display).

The guide unit 11 c is a constituent unit including a control circuit,for example, configured to generate guide information for performingguide display relating to a performed operation, guide display for anoperation to be next performed, and the like based on the image dataacquired by the two image pickup apparatuses (20, 30) and theinformation stored in the database 13 in advance.

The object judgment unit 11 d and the size judgment unit 11 e are each aconstituent unit including a control circuit, for example, configured todetect and judge, based on the image data acquired by the two imagepickup apparatuses (20, 30) and the information stored in the database13 in advance, predetermined conditions such as a type, a site, a shape,a size, a number, and a state of a predetermined object included in theimage data. Note that as size judgment means, means for calculating thesize of an object formed on a light receiving surface of an image pickupdevice from a relationship between a distance between the image pickupapparatus and the object and a size of the image pickup device using anautomatic focus mechanism, for example, can also be used in addition tothe above.

The communication unit 12 is a transmission unit and a receiving unitincluding a control circuit, for example, configured to perform datacommunication (transmission and receiving of information) between therespective communication units (22, 32) in the two image pickupapparatuses (20, 30). The communication unit 12 may have a configurationusing wireless communication or a configuration using wiredcommunication.

The database 13 is a storage device (auxiliary storage device) storingvarious types of information determined in advance. In the presentembodiment, an example in which the database 13 is included in thecontrol apparatus 10 (more specifically, an internally fixed storagemedium (a built-in memory, etc.) or a removable storage medium (aremovable card-type memory, etc.), for example) is illustrated. However,the configuration of the database 13 is not limited to such forms. Forexample, the database may be configured by an external storage device inan independent form. Note that in this case, an area between thedatabase having an independent configuration and the control apparatus10 may be configured to enable transmission and receiving of varioustypes of data using predetermined connection means by wireless or wire.

The database 13 stores an operation information database 13 a and animage database 13 b, for example. The operation information database 13a includes various types of information about a plurality of operations(object information, used tool information, operation type information,etc.). The image database 13 b is a collection of image data associatedwith various types of information recorded on the operation informationdatabase 13 a (e.g., image information from a plurality of visual fieldsrelating to a plurality of operations) (details are described below; seeFIG. 12).

The evidence recording unit 14 is a constituent unit including a controlcircuit and the like and a storage medium and the like each configuredto record the image data and the like acquired by the two image pickupapparatuses (20, 30) and operation information associated with the imagedata in association with each other. Needless to say, images from morevisual fields, fields of view, and viewpoints may be enabled to bereferred to.

The first image pickup apparatus out of the two image pickup apparatuses(20, 30) is the wearable terminal apparatus 20. The wearable terminalapparatus 20 is an image pickup apparatus in such a form that frees bothhands of the operator by wearing the wearable terminal apparatus 20 on apart of the body (e.g., the vicinity of the face) and at the same time astate in the vicinity of the hands of the operator can be always pickedup, as illustrated in FIG. 2, for example.

In other words, the wearable terminal apparatus 20 is an image pickupapparatus in a form capable of acquiring operation informationcorresponding to the eye line of the operator.

The wearable terminal apparatus 20 mainly acquires the movie dataserving as the first image data. The movie image data is used forgrasping a frequent flow of operations by the operator, and may be astill image in which a time change can be found or data information,representing a time change of an image, such as a motion vectorextracted from the still image. The movie image data is preferably astill image, which can be verified later by a person, when used forevidence.

FIG. 2 is a conceptual diagram illustrating a schematic configuration ofthe first image pickup apparatus (the wearable terminal apparatus)constituting a part of the operation support system according to thepresent embodiment. Note that FIG. 2 supposes a state where the operatoris wearing the first image pickup apparatus (wearable terminalapparatus). FIG. 3 is a conceptual diagram illustrating a frontal viewof an operator's face with the operator wearing the first image pickupapparatus (the wearable terminal apparatus) and simultaneouslyillustrating a field of view by both the eyes of the operator. FIG. 4 isa conceptual diagram more specifically illustrating a field of view ofboth the eyes of the operator illustrated in FIG. 3.

In the present embodiment, the wearable terminal apparatus 20 is animage pickup apparatus which individually functions under the control ofa control unit 21, and simultaneously also functions under the controlof the control unit 11 in the control apparatus 10 by communicating withthe control apparatus 10.

Note that, while the wearable terminal apparatus 20 according to thepresent embodiment is an image pickup apparatus configured to mainlyacquire movie data, the present invention is not limited to this. Forexample, the wearable terminal apparatus 20 may be configured to be ableto optionally or automatically acquire still image data simultaneouslywith movie data or independently acquire still image data using apredetermined operation by the operator or using a predetermined actionas a trigger.

The wearable terminal apparatus 20 includes the control unit 21, thecommunication unit 22, an operation unit 23, an image pickup unit 24 (afirst image pickup unit), a display unit 25, a voice input unit 27, avoice output unit 28, a support unit 29, and the like.

The control unit 21 is a constituent unit including a control circuit,for example, configured to integrally control the entire wearableterminal apparatus 20. The control unit 21 is composed of a processorusing a CPU (central processing unit), for example. The control unit 21controls each of the constituent units by operating according to aprogram stored in a memory (not illustrated).

That is, the control unit 21 performs various types of control, forexample, controls the communication unit 22 to transmit and receive(communicate) various types of information to and from the controlapparatus 10, controls each of the constituent units upon receipt of anoperation input from the operation unit 23, controls the image pickupunit 24 to acquire image data and the like, controls the display unit 25to display various types of information on a screen of a display panel25 b (described below; see FIG. 2), controls the voice input unit 27 toreceive an input of voice information, and controls the voice outputunit 28 to output voice information.

In addition to the various types of control, the control unit 21controls the wearable terminal apparatus 20 in cooperation with thecontrol apparatus 10 upon receipt of a control signal from the controlunit 11 in the control apparatus 10.

The control unit 21 includes a plurality of circuit units such as animage judgment unit 21 b and a display control unit 21 c.

The image judgment unit 21 b is a judgment processing circuit configuredto perform predetermined judgment based on the image data acquired bythe image pickup unit 24. The image judgment unit 21 b is a circuit unitwhich is equivalent to the various types of judgment units (11 a, 11 d,and 11 e) included in the control unit 11 in the control apparatus 10 oris simple.

That is, the image judgment unit 211 b functions as an operationjudgment unit configured to judge the first image data and the secondimage data according to the operation information in the database 13.Therefore, if the control unit 11 in the control apparatus 10 isprovided with the various types of judgment units (11 a, 11 d, and 11e), the image judgment unit 21 b can also be omitted.

If the image judgment unit 21 b is provided on the side of the wearableterminal apparatus 20, only a judgment result by the image judgment unit211 b may be transmitted to the control unit 11 in the control apparatus10 via the communication units (22, 12). The control unit 11 in thecontrol apparatus 10 performs control based on received judgment resultinformation upon receipt of the judgment result. Therefore, theconfiguration has the advantage that a communication load can bereduced.

The display control unit 21 c is a control circuit unit configured tocontrol the display unit 25 to perform predetermined display on thedisplay panel 25 b (see FIG. 2).

The communication unit 22 is a transmission unit and a receiving unitincluding a control circuit, for example, configured to performcommunication (transmission and receiving of information) with thecontrol apparatus 10. Predetermined communication means by wireless orwire is applied to the communication unit 22. The communication unit 22also functions as a communication unit configured to acquire the secondimage data from the second image pickup apparatus (the auxiliary camera30) provided at a position different from a position of the operator 100(or the wearable terminal apparatus 20 serving as the first image pickupapparatus) via the control apparatus 10.

Further, the communication unit 22 can also refer to the operationinformation in the database 13 by communicating with the database 13 viathe control apparatus 10. Note that the communication unit 22 and thedatabase 13 may directly communicate with each other.

The operation unit 23 is a constituent unit (a specific configuration ofthe constituent unit is not illustrated) including a plurality ofoperation members configured to operate the wearable terminal apparatus20 and a plurality of electrical components respectively correspondingto the plurality of operation members and configured to each generate apredetermined operation input signal.

The image pickup unit 24 is a constituent unit including an image pickupoptical system 24 a (see FIG. 2) and an image pickup device and acontrol circuit for driving the image pickup device (not illustrated).The image pickup unit 24 has functions, for example, of sequentiallyphotoelectrically converting an optical image to be picked up formed bythe image pickup optical system 24 a using the image pickup device andsequentially displaying an image based on image data thus obtained onthe display panel 25 b in the display unit 25. The image pickup unit 24has functions, for example, of forming image data (e.g., movie data orstill image data) in a predetermined form based on an obtained imagesignal, recording the image data, and transmitting the image data to thecontrol apparatus 10 via the communication units 22, 12. A configurationof the image pickup unit 24 having such functions is substantiallysimilar to a configuration of an image pickup unit in a conventionallygeneral image pickup apparatus. Therefore, a specific configuration andillustration of the image pickup unit 24 are omitted.

Note that photoelectric conversion elements such as a CCD (chargecoupled device) image sensor and a CMOS (complementary metal oxidesemiconductor) image sensor are applied as the image pickup device. Inthe image pickup unit, an image close to an eye line (a field of viewand a visual field) of an operator in operation is obtained, and aspecific operation (including a visual check) using both hands performedbetween both the hands is easily confirmed.

The display unit 25 is a constituent unit including a display controlcircuit, for example, configured to enable various types of warningdisplay or various types of predetermined message display or guidedisplay (details are described below) by overlapping the display with animage in addition to displaying an image based on the image dataacquired by the image pickup unit 24 while performing menu display andthe like for various types of settings in the wearable terminalapparatus 20 by switching a predetermined operation mode, as needed.

The display unit 25 includes a light guiding unit 25 a, a display panel25 b, and the like. The display panel 25 b is a constituent unitincluding a display screen on which various types of display are to beperformed and a display control circuit or the like for implementing thedisplay. An LCD (liquid crystal display) unit and an OEL (organicelectro-luminescence) display unit, for example, are applied as thedisplay panel 25 b.

The light guiding unit 25 a is a constituent unit configured to guide animage displayed on the display panel 25 b to a predetermined displaysurface.

The display panel 25 b in the wearable terminal apparatus 20 accordingto the present embodiment is installed to be directed toward a differentdirection from a direction of a surface opposing an eye direction (adirection indicated by an arrow X1 in FIG. 2) of the operator, forexample, as illustrated in FIG. 2.

More specifically, the eye direction of the operator is indicated by anarrow X1, and a direction toward which the display surface of thedisplay panel 25 b is directed is indicated by an arrow X2 in FIG. 2,for example. In this case, the light guiding unit 25 a is provided suchthat a direction of the long axis matches the direction indicated by thearrow X2 in FIG. 2. Therefore, light forming an image on the displaysurface of the display panel 25 b is guided toward the directionindicated by the arrow X2 by the light guiding unit 25 a.

In the light guiding unit 25 a, a light bending member (a prism, etc.)configured to bend the above-described light guided toward the directionindicated by the arrow X2 by the light guiding unit 25 a and guide thebent light in the direction indicated by the arrow X1 is disposed at oneend (see a reference sign 25 x) of the light guiding unit 25 a.Therefore, light emitted from the display surface of the display panel25 b is guided toward the direction indicated by the arrow X2 in thelight guiding unit 25 a, and is displayed at a position (see a referencesign 25 c in FIG. 3) which can be visually recognized by one of the eyesof the operator via the prism or the like.

The voice input unit 27 is a constituent unit including an electricalcomponent configured to input voice information, a driving circuitconfigured to drive the electrical component, and the like. The voiceinput unit 27 is controlled by the control unit 21, to collect voiceinformation under an ambient environment at a predetermined timing. Asthe voice input unit 27, a microphone, for example, is applied.

As an arrangement of the voice input unit 27, the voice input unit 27 isdesirably arranged in the vicinity of the mouth of the operator when theoperator wears the wearable terminal apparatus 20, for example (see FIG.2). Such an arrangement enables the operator to reliably perform a voiceinput to the voice input unit 27.

The voice output unit 28 is a constituent unit including an electricalcomponent configured to output voice information, a driving circuitconfigured to drive the electrical component, and the like. The voiceoutput unit 28 is controlled by the control unit 21, to output voiceinformation (e.g., voice information and a voice guide such as a warningand a message) at a predetermined timing. As the voice output unit 28, asound production device such as a speaker is applied.

As an arrangement of the voice output unit 28, the voice output unit 28is desirably arranged in the vicinity of the ear of the operator whenthe operator wears the wearable terminal apparatus 20, for example (seeFIG. 2). Such an arrangement enables the operator to reliably catch avoice output from the voice output unit 28.

The support unit 29 (see FIG. 2) is a frame member configured such thateach of the constituent units in the wearable terminal apparatus 20 isassembled thereinto while being wearable on a part of the body (e.g.,the vicinity of the face) of the operator. The configuration example ofthe support unit 29 illustrated in FIG. 2 is an example in which thesupport unit 29 is formed in a so-called glass frame shape.

Note that the wearable terminal apparatus 20 in the present embodimentincludes a battery serving as a power source, although not illustrated.The battery generates electric power required to drive the wearableterminal apparatus 20, and is controlled by the control unit 21, tosupply the electric power to each of the constituent units.

FIGS. 2 and 3, for example, illustrate how the operator wears thewearable terminal apparatus 20 thus configured.

In FIG. 3, the right eye and the left eye of the operator 100 arerespectively denoted by a reference sign 101R and a reference sign 101L.In FIGS. 3 and 4, a right eye visual field by the right eye 101R and aleft eye visual field by the left eye 101L are respectively indicated bya dotted frame denoted by a reference sign 110R and a dotted framedenoted by a reference sign 110L.

In this case, the operator 100 wears the support unit 29 in the wearableterminal apparatus 20 on the vicinity of the right eye (of the operator100), for example.

Thus, when the wearable terminal apparatus 20 is being worn, an imagepickup optical system 24 a in the image pickup unit 24 is directedtoward a direction which substantially matches the eye direction of theoperator (the direction indicated by the arrow X1). When the eyedirection is changed by turning the head of the operator 100, forexample, with the wearable terminal apparatus 20 worn, a directiontoward which the image pickup optical system 24 a in the image pickupunit 24 is directed is changed by following the eye direction.Therefore, image data acquired by the image pickup unit 24 with thewearable terminal apparatus 20 worn by the operator 100 becomes apredetermined area including a direction in which the operator is seeingthe image data (an eye direction). An image pickup area of the imagedata acquired by the image pickup unit 24 is a frame line denoted by areference sign 24 c in FIG. 4. The image pickup area 24 c can beoptionally set by an angle of view of the image pickup optical system 24a. In the present embodiment, the image pickup area 24 c is set to be anarea including the right eye visual field 110R and the left eye visualfield 110L of the operator 100.

Further, with the wearable terminal apparatus 20 worn by the operator100, display by the display panel 25 b in the display unit 25 isperformed in a predetermined area denoted by a reference sign 25 c inFIGS. 3 and 4 within the right eye visual field 110R of the operator 100via the light guiding unit 25 a. The display area 25 c is an area insuch a form as to be visually recognizable in a part of the whole ofboth the right and left visual fields (110R, 110L) of the operator 100and formed enough not to inhibit the visual field of the operator 100.The operator 100 can visually recognize the whole of both the right andleft visual fields (110R, 110L) while closely observing the display area25 c by performing eye movement and visually recognizing a displaycontent of the display unit 25.

On the other hand, the other second image pickup apparatus out of thetwo image pickup apparatuses (20, 30) is the auxiliary camera 30. Theauxiliary camera 30 is an image pickup apparatus in a general form or animage pickup apparatus placed or fixedly installed at a predeterminedposition in the vicinity of an operation table (not illustrated).

In other words, the auxiliary camera 30 is an image pickup apparatus insuch a form as to be able to acquire operation information differentfrom the operation information corresponding to the eye line of theoperator.

The auxiliary camera 30 preferably has information about a so-calledobjective standpoint from a visual field different from a visual fieldof the image through the eye line of the operator (becomes rich ininformation as a system), and mainly acquires the still image dataserving as the second image data. When the auxiliary camera 30 is usedfor evidence, a still image, which can be verified later by a person, ispreferable. However, the still image becomes a movie if continuouslyacquired. Therefore, the auxiliary camera 30 may be thus applied if ithas no problem even in consumed energy and a recording capacity.

A configuration of the auxiliary camera 30 is substantially similar to aconfiguration of an image pickup apparatus in a general form. Forexample, the auxiliary camera 30 includes at least the control unit 31,the communication unit 32, an image pickup unit 34 (a second imagepickup unit), and the like, as illustrated in FIG. 1.

The control unit 31 is a constituent unit including a control circuit,for example, configured to integrally control the entire auxiliarycamera 30. The control unit 31 is composed of a processor using a CPU,for example. The control unit 31 controls each of the constituent unitsby operating according to a program stored in a memory (notillustrated).

The control unit 31 performs various types of control. For example, thecontrol unit 31 controls the communication unit 32 to transmit andreceive (communicate) various types of information to and from thecontrol apparatus 10, and controls the image pickup unit 34 to acquireimage data and the like. In addition to the various types of control,the control unit 31 controls the auxiliary camera 30 in cooperation withthe control apparatus 10 upon receipt of a control signal from thecontrol unit 11 in the control apparatus 10.

The communication unit 32 is a transmission unit and a receiving unitincluding a control circuit, for example, configured to performcommunication (transmission and receiving of information) with thecontrol apparatus 10. Predetermined communication means by wireless orwire is applied to the communication unit 32.

Note that the communication unit 32 can also refer to the operationinformation in the database 13 by communicating with the database 13 viathe control apparatus 10. The communication unit 32 and the database 13may directly communicate with each other.

The image pickup unit 34 is a constituent unit including an image pickupoptical system and an image pickup device and a control circuit fordriving the image pickup device (not illustrated). The image pickup unit34 has functions such as a function of sequentially photoelectricallyconverting an optical image of an object to be picked up formed by theimage pickup optical system using the image pickup device andtransmitting image data thus obtained to the control apparatus 10 viathe communication unit 32. The image pickup unit 34 simultaneously has afunction of sequentially displaying an image based on the image datausing a display unit (not illustrated).

A configuration of the image pickup unit 34 having such functions issubstantially similar to the configuration of the above-describedwearable terminal apparatus 20 or the image pickup unit in theconventionally general image pickup apparatus. For example, the imagepickup unit 34 also includes constituent members (an operation unit, adisplay unit, a battery, etc.) included in an image pickup apparatus ina conventionally general form, illustration of which is omitted inFIG. 1. The constituent members each have a similar configuration andfunction to a configuration and function of the image pickup apparatusin the conventionally general form, and hence detailed description isomitted.

Note that the control unit 31 may also be provided with an imagejudgment unit (operation judgment unit) in the auxiliary camera 30. Ifthe control unit 31 in the auxiliary camera 30 is provided with theimage judgment unit, the auxiliary camera 30 may transmit only ajudgment result by the image judgment unit to the control unit 11 in thecontrol apparatus 10 via the communication units (32, 12), like when thecontrol unit 21 in the wearable terminal apparatus 20 is provided withthe image judgment unit 21 b. Upon receipt of the judgment result, thecontrol unit 11 in the control apparatus 10 performs control based oninformation about the received judgment result. Therefore, theconfiguration has the advantage that a communication load can bereduced.

The auxiliary camera 30 thus configured is fixedly arranged in thevicinity of an operation place when the operator performs apredetermined operation. In this case, the auxiliary camera 30 isarranged at a position where an object to be an operation target and thevicinity of the hand of the operator who handles the object are includedin at least the image pickup area. Further, the auxiliary camera 30 maybe more preferably arranged such that an entire image of the operator isincluded in the image pickup area to be able to confirm how the operatoris operating. Thus, a different image from an image through the eye lineof the operator is obtained.

Functions performed using the operation support system 1 according tothe present embodiment thus configured will be described below.

Although various types of operations in various types of fields can beassumed as an operation performed using the operation support system 1according to the present embodiment, a maintenance operation relating towashing or the like of a predetermined medical instrument amongoperations for handling medical instruments will be described below as aspecific example of the operation in the following description. Morespecifically, an operation for washing and sterilizing a surgicalinstrument or a treatment tool (e.g., an instrument in the shape ofscissors, which is abbreviated as scissors) serving as a medicalinstrument already used (hereinafter referred to as an instrumentwashing and sterilization operation) is illustrated.

First, a schematic flow of an instrument washing and sterilizationoperation generally performed will be simply described with reference toa flowchart of FIG. 5. FIG. 5 is a flowchart illustrating a flow ofoperations performed by an operator in a maintenance operation (aninstrument washing and sterilization operation) for a medicalinstrument.

In step S1101, an operator first confirms an object (an instrument)serving as a maintenance operation target in a normal instrument washingand sterilization operation, as illustrated in FIG. 5.

In step S1102, the operator then confirms a state of a maintenanceenvironment. The maintenance environment is an environment in a location(a room, a table, etc.) where the operation is to be performed. Theconfirmation of the state of the environment is confirmation whether theenvironment is in a clean state, i.e., whether the environment is in astate appropriate for the instrument washing and sterilizationoperation. In a normal case, a dedicated maintenance environment (aroom, a table, etc.) is prepared.

In step S1103, the operator then performs a first operation in a firstprocess in the maintenance operation.

The maintenance operation is considered as a series of operations whichis established when a plurality of processes each serving as acollection of a plurality of operations are performed. That is, the onepredetermined maintenance operation is established when the plurality ofoperation processes are executed in a predetermined procedure. Each ofthe plurality of operation processes further includes a plurality ofspecific operational actions. When the plurality of operations areperformed in a predetermined procedure, each of the operation processesis established.

Examples of the first operation in the first process include anoperation for confirming a maintenance tool in step S1103. Examples ofthe maintenance tool include a washing brush for washing an instrument.In this case, a plurality of types of washing brushes are prepared asthe maintenance tool. Appropriate tools are prepared to be optimized foreach maintenance target instrument or depending on a target site in apredetermined instrument, and the most appropriate tool needs to beselected for each of the processes in the maintenance operation and foreach of the operations. Thus, a used tool needs to be confirmed beforeeach of the operations.

In step S1104, the operator then performs a second operation in thefirst process. Note that, while the second operation is an operation forwashing a predetermined first site of a maintenance target instrument(e.g., a specific site such as a blade portion of scissors), forexample, specific illustration of the second operation is omitted.

In step S1105, the operator then performs a third operation in the firstprocess. Note that the third operation is similarly an operation forwashing a predetermined second site different from the first site of themaintenance target instrument (e.g., a specific site such as a jointportion of scissors), for example, specific illustration of the thirdoperation is omitted.

In step S1106, the operator then confirms whether the plurality ofoperations defined in the first process have ended. In an example of theoperations illustrated in the flowchart of FIG. 5, the first process iscompleted in the first to third operations.

Therefore, when the operations in the first process have not ended atthis time point, the processing returns to step S1104, described above.In step S1104 and subsequent steps, similar processing is repeated. Whenit is confirmed that all the operations in the first process have ended,the processing proceeds to step S1107, described below.

In step S1107, the operator then performs evidencing processing relatingto each of the operations in the first process. The evidencingprocessing is an operation for the operator to confirm that each of theoperations in the first process has been completed and clearly indicatethe confirmation as evidence. More specifically, examples of theevidencing processing include confirmation whether the operator hasexecuted operation items to be confirmed which are listed in advance foreach of the operations and creation of a check list to be recorded.

In step S1108, the operator then confirms whether all the operationprocesses in the maintenance operation (the instrument washing andsterilization operation) have ended. If it is confirmed that all theoperation processes have not ended, the processing proceeds to stepS1110. If it is confirmed that all the operation processes have ended,the processing proceeds to step S1109.

In step S1110, the operator performs an operation for changing amaintenance tool which the operator currently has into a maintenancetool used in the subsequent operation process. The processing returns tostep S1103, described above.

On the other hand, if it is confirmed that all the operation processeshave ended, the processing proceeds to step S1109. In step S1109, theoperator performs final evidencing processing for the maintenanceoperation (the instrument washing and sterilization operation). Examplesof the final evidencing processing include an operation for compilingcheck lists respectively created for the processes while creating acomprehensive and final operation check list and compiling the checklist as evidence information.

The above-described evidence information is a document in a list formatinto which a plurality of information are compiled in a predeterminedform, as illustrated in FIG. 6, for example. Note that the evidenceinformation may be in a document format in such a form as to be printedon a sheet surface or in a so-called electronic file format in anelectronic form, for example. FIG. 6 is a diagram illustrating anexample of evidence information generated by a maintenance operation.

Examples of the evidence information include a data collection in a listformat obtained by listing information (index information) aboutinstruments to be maintained, a plurality of operation processesdetermined for each of the instruments to be maintained, informationabout a plurality of operation contents to be performed in each of theoperation processes, a plurality of confirmation items corresponding toeach of the operation contents, and the like.

Examples of the plurality of confirmation items included in the evidenceinformation include an ambient environment (e.g., an operation room, anoperation table, etc.) in which an operation is performed, a name of atool used for the operation, a name of an operator, and a time period (adate, a start/end time, etc.) for the operation. The plurality ofconfirmation items are determined in advance for each of the pluralityof operations in each of the operation processes. In addition to a listof the confirmation items, a plurality of image (still image and movie)data acquired during each of the operations are recorded as evidenceinformation, and image data respectively corresponding to theconfirmation items are recorded in association with one another.

Particularly, a still image is easily left in writing and easily kept asa material, and is thus important as evidence. By movie information,evidence in a still image at an optimum timing is obtained. When moviedata in an image pickup range substantially equivalent to a field ofview of the operator is acquired, the operator can detect meticulousmovement while confirming the movie data. That is, the operation can beconfirmed based on grasp of a situation including an intension of theoperator and a confidence in the operation.

Therefore, a wearable terminal apparatus (first image pickup apparatus)to be worn by the operator is provided, and an auxiliary camera (secondimage pickup apparatus) for acquiring, to acquire information, which isinsufficient by only shooting in the same direction as a direction ofthe wearable terminal apparatus, still image data in a predeterminedimage pickup range including a predetermined object in a differentdirection from the direction, to increase an amount of information andcause evidence of the information amount to have diversity. That is,there can be provided an operation support system including a controlapparatus configured to control the first image pickup apparatus and thesecond image pickup apparatus in cooperation with each other whileacquiring operation information about a predetermined operation andconfigured to control the auxiliary camera to acquire still image dataserving as evidence of the operation according to movie data as almostviewed by the operator.

Conventionally, image data or the like recorded as evidence informationhas been acquired by a manual operation of an operator every time in aseries of operations, each of the operations is completed. In theoperation support system 1 according to the present embodiment, theevidence information including the image data or the like is configuredto be automatically recorded.

Functions at the time when the operation support system 1 according tothe present embodiment is used will be described below. FIG. 7 is aconceptual diagram illustrating how a predetermined maintenanceoperation (instrument washing and sterilization operation) is beingperformed using the operation support system according to the presentembodiment.

As illustrated in FIG. 7, when the maintenance operation (instrumentwashing and sterilization operation) illustrated as an example in thepresent embodiment is performed, an operator 100 performs an operationwhile sitting on a chair 151 at an operation table 150 installed withina predetermined operation room. At this time, the operator 100 remainswearing the wearable terminal apparatus 20 in the vicinity of the face.Thus, the wearable terminal apparatus 20 acquires image data (mainlymovie data) in a predetermined area including an eye direction of theoperator 100.

On the operation table 150, the auxiliary camera 30 is placed to bedirected toward the operator 100. The auxiliary camera 30 acquires imagedata in a predetermined area including at least the vicinity of the handof the operator 100. Note that the auxiliary camera 30 desirably has awaterproof function in consideration of use in plumbing.

Illustration of the control apparatus 10, which cooperates with the twoimage pickup apparatuses (20, 30), is omitted in FIG. 7. The reason isthat the control apparatus 10 is installed in a distant place because anoperation place for the operation illustrated as an example in thepresent embodiment is in the vicinity of plumbing.

The maintenance operation (instrument washing and sterilizationoperation) herein illustrated as an example is an operation for washingand sterilizing scissors 200 serving as a surgical instrument, atreatment tool, or the like serving as a used surgical instrument.Therefore, the operation table 150 includes a washing tank 150 a.Further, a faucet 153 is installed above the washing tank 150 a. When ahandle 153 a in the faucet 153 is operated to rotate in an openingdirection, running water 154 flows out of a running water port 153 b.

The operator 100 brushes and washes an instrument (the scissors 200) tobe maintained using a predetermined tool (the washing brush 300) in awashing liquid (e.g., water) stored in the washing tank 150 a.

In this case, image data acquired by the wearable terminal apparatus 20becomes an image as illustrated in FIG. 8, for example. FIG. 8illustrates a display example of an image based on image data acquiredby the wearable terminal apparatus during the maintenance operation(instrument washing and sterilization operation) using the operationsupport system according to the present embodiment.

The display example of the image illustrated in FIG. 8 indicates how theoperator 100 holds the maintenance target instrument (the scissors 200)in hand (left hand 100L) and is performing a brushing and washingoperation using the predetermined tool (the washing brush 300) in theother hand (right hand 100R). At this time, the wearable terminalapparatus 20 picks up an area in the vicinity of the hand of theoperator 100.

Functions at the time when a predetermined maintenance operation(instrument washing and sterilization operation) is performed will bedescribed below using the operation support system 1 according to thepresent embodiment.

As described above, the operation support system 1 according to thepresent embodiment is a system which functions by causing the two imagepickup apparatuses (the wearable terminal apparatus 20 and the auxiliarycamera 30) to cooperate with each other when the operator 100 performs apredetermined operation.

The outline of the functions of the operation support system 1 accordingto the present embodiment will be described below.

In the operation support system 1 according to the present embodiment,the two image pickup apparatuses (20, 30) each pick up a situation of aplurality of operations sequentially performed by the operator 100, asneeded. Each of image data thus acquired is transmitted to the controlapparatus 10.

The control apparatus 10 receives the image data respectively acquiredby the two image pickup apparatuses (20, 30), and performs predeterminedjudgment processing based on the image data and various types ofinformation stored in a database in advance. Examples of the judgmentprocessing performed by the control apparatus 10 include a plurality ofconfirmation items defined depending on a target operation content,e.g., judgment of a maintenance target instrument, judgment of the typeof a used tool, and judgment of an operational action (brushing,washing, scraping, agent spraying, etc.).

The control apparatus 10 performs various types of control processingbased on the above-described judgment processing result. For example, ifit is judged as a judgment result that the performed operational actionmatches a defined operation (is a correct operation), the controlapparatus 10 performs evidence recording in addition to creating aconfirmation check table, for example. Examples of the evidencerecording include processing for recording information included in theevidence information illustrated in FIG. 6, for example.

If it is judged as a judgment result that the performed operationalaction is out of the defined operation (is an operation error), warningdisplay indicating that the operational action is out of the definedoperation, for example, is performed while guide display for performingthe correct operation again, for example, is performed.

Note that a specific information content included in the operationinformation database 13 a is illustrated as an example in FIG. 12. Theexample illustrated in FIG. 12 is an example of a database assuming amaintenance operation performed when “scissors” serving as one type ofinstrument used for surgery, treatment, or the like are washed using amanual operation of an operator after being used by surgery, treatment,or the like (abbreviated as a scissors washing and sterilizationoperation below).

As illustrated in FIG. 12, in the operation information database, amaintenance target column [A] is a name of a maintenance targetinstrument. In this example, information specified as “target 1:scissors” is recorded, and the information indicates a maintenanceoperation of scissors washing and sterilization. Note that, while a nameof another instrument enters “target 2:000”, detailed illustration otherthan information about the “target 1” is omitted in FIG. 12.

In the operation information database illustrated in FIG. 12, anoperation process column [B] lists examples of a plurality of operationprocesses included in the maintenance operation in the order ofoperations. Examples of the operation processes included in the scissorswashing and sterilization operation include

“first process: brushing” process,

“second process: running water washing” process,

“third process: scraping” process, and

“fourth process: agent sterilization” process.

The “first process: brushing” process is a process for brushing a targetinstrument (here, the scissors) in a predetermined water tank or thelike using a predetermined tool (e.g., a brush).

The “second process: running water washing” process is a process forwater-washing the scissors in running water.

The “third process: scraping” process is a process for scraping thescissors using a predetermined tool (e.g., paper or a cloth) to wipewater or dirt.

The “fourth process: agent sterilization” process is a process for(spraying and) sterilizing the scissors using a predetermined agent(e.g., alcohol).

An example herein illustrated is an example of a “scissors washing andsterilization operation”. Therefore, the respective types of and thenumber of processes to be performed differ depending on a targetinstrument.

In the operation information database illustrated in FIG. 12, anoperation content column [C] lists examples of a plurality of operationsincluded in each of the operation processes in the order of operations.

Examples of operations included in the “first process: brushing” processin the scissors washing and sterilization operation include

“tool confirmation” operation serving as a “first operation”,

“first site brushing” operation serving as a “second operation, and

“second site brushing” operation serving as a “third operation”.

Specific information about each of the operations is recorded in a nextoperation information column “D”.

That is, in the operation information database illustrated in FIG. 12,the operation information column [D] records specific information (e.g.,character data or image data) for clearly indicating each of theoperations. Note that, while the image database 13 b is separatelyprovided in addition to the operation information database 13 a in FIG.1, both the operation information database 13 a and the image database13 b may be integrally configured as the configuration of the database13.

The information in the operation information column [D] further includeswearable terminal apparatus information [D1] obtained using the firstimage pickup apparatus (wearable terminal apparatus 20) and auxiliarycamera information [D2] obtained using the second image pickup apparatus(auxiliary camera 30). The wearable terminal apparatus information [D1]is operation information mainly represented by a movie. The auxiliarycamera information [D2] is operation information mainly represented by astill image.

The “tool confirmation” operation serving as the “first operation”included in the “first process: brushing” process is an operation forconfirming a tool used in the operation process. Therefore, in theoperation information [D], movie data (e.g., data referred to asmovement data), for example, in which an action for holding the tool(brush) over a front surface (within an image pickup range) of theauxiliary camera 30 is recorded as the wearable terminal apparatusinformation [D1].

Examples of the auxiliary camera information [D2] include an enlargedimage of the tool (brush) and an image of an ambient environment of theoperator 100 (described as a “peripheral image” in FIG. 12). As theperipheral image, information about an operation room, an operationtable, or the like, or clothes of the operator 100 is recorded as datasuch as text data or image data.

When the operator performs the “tool confirmation” operation,operations, described below, are performed. That is, the operator 100first performs an action for holding the predetermined tool (washingbrush 300) used for an operation to be now performed over the frontsurface (within the image pickup range) of the auxiliary camera 30 withthe predetermined tool in hand. The action is recorded as a movie by thewearable terminal apparatus 20. FIG. 13 is a diagram illustrating asituation at this time. That is, FIG. 13 illustrates one scene of themovie acquired by the wearable terminal apparatus 20 when the operator100 has held the predetermined tool (the washing brush 300), which theoperator 100 has in right hand 100R, over the front surface (within theimage pickup range) of the auxiliary camera 30 to perform the “toolconfirmation” operation. Note that a portion indicated by dotted hatchedlines in FIG. 13 indicates how a washing liquid such as water is storedin the washing tank 150 a. Brushing or the like of an instrument ismainly performed in the washing liquid such as water stored in thewashing tank 150 a.

The wearable terminal apparatus 20 picks up the action (the action forholding the washing brush 300 over the front surface of the auxiliarycamera 30) performed by the operator 100. Simultaneously, the auxiliarycamera 30 also picks up a live view image when an instrument (tool) in adefined form enters the image pickup range. Both the image pickupapparatuses (20, 30) each always continuously transfer acquired moviedata (live view image data) to the control apparatus 10.

The control apparatus 10 performs predetermined image judgmentprocessing based on the received movie data. When the above-describeddefined action (the action for holding the brush over) is detected, forexample, the control apparatus 10 controls the auxiliary camera 30, toperform an operation for picking up a still image. Thus, an enlargedimage (still image) of the used tool (the washing brush 300) isrecorded. The still image acquired at this time is as illustrated inFIG. 14, for example.

Further, subsequently thereto or simultaneously therewith, the auxiliarycamera 30 records as a still image a state of an ambient environment(e.g., an operation room or an operation table) including the operator100 (not illustrated).

Image data or the like thus acquired contributes to judgment whether theused tool (brush) is correct for use in the operation or whether theperformed operational action is correct by referring to the image dataor the like stored in the database 13. If it is judged that theperformed action is correct as a result of the judgment, the acquiredimage data or the like is recorded as evidence information.

The “first site brushing” operation serving as the “second operation”included in the “first process: brushing” process is an operation forbrushing a first site of the target instrument “scissors”. The firstsite is a “blade portion”. Examples of operation information in thiscase include a number of times of brushing and a brushing time perioddetected based on an operation image, which are put into a database, asthe wearable terminal apparatus information [D1], and a partiallyenlarged image of the first site of the tool (brush) as the auxiliarycamera information [D2]. Note that FIG. 15 illustrates one scene of themovie data acquired by the wearable terminal apparatus 20 when the firstsite (blade portion) is being brushed. The operation is performed inwater with which the washing tank 150 a is filled. A portion indicatedby dotted hatched lines in FIG. 15 indicates a washing liquid such aswater stored in the washing tank 150 a. That is, FIG. 15 illustrates howthe instrument (scissors 200) is being brushed with the washing brush300 in washing liquid such as water.

Similarly, the “second site brushing” operation serving as the “thirdoperation” included in the “first process: brushing” process is anoperation for brushing a second site of the target instrument“scissors”. The second site is a “joint portion”. Operation informationin this case is substantially similar to the operation information inthe above-described “second operation”. That is, examples of theoperation information include a number of times of brushing and abrushing time period, which are put into a database, as the wearableterminal apparatus information [D1], and a partially enlarged image ofthe second site of the tool (brush) as the auxiliary camera information[D2]. Note that FIG. 8 illustrates one scene of the movie data acquiredby the wearable terminal apparatus 20 when the second site (jointportion) is being brushed.

Note that in the “second process: running water washing” process, afaucet of tap water, for example, is confirmed as a tool while an actionfor opening the faucet is confirmed and a flow rate or the like of tapwater (running water) flowing from the faucet is confirmed. In theoperation information, a defined operation time period for the targetsite is recorded.

In the “third process: scraping” process, examples of confirmation of adefined dedicated scraper such as paper or a cloth used for a scrapingoperation include confirmation of a brand of the scraper using an imageof a package or the like of the scraper. In the operation information, adefined number of times of operation, operation time period, and thelike for the target site are recorded.

In the “fourth process: agent sterilization” process, a brand or thelike of an agent used for a sterilization operation, for example, adefined agent such as alcohol is confirmed using an image of a bottle orthe like of the agent. In the operation information, a defined number oftimes of operation, operation time period, and the like for the targetsite are recorded.

Functions at the time when a predetermined maintenance operation(instrument washing and sterilization operation) is performed using theoperation support system 1 according to the present embodiment will bedescribed below with reference to flowcharts illustrated in FIGS. 9 to11.

FIG. 9 is a flowchart illustrating functions of the control apparatus inthe operation support system according to the present embodiment. FIG.10 is a flowchart illustrating functions of the wearable terminalapparatus (the first image pickup apparatus) in the operation supportsystem according to the present embodiment. FIG. 11 is a flowchartillustrating functions of the auxiliary camera (the second image pickupapparatus) in the operation support system according to the presentembodiment.

First, in the operation support system 1 according to the presentembodiment, power to each of the apparatuses (the control apparatus 10,the wearable terminal apparatus 20, and the auxiliary camera 30) isturned on. In this state, the operator 100 starts a predeterminedoperation under a predetermined environment illustrated in FIG. 7. Asthe predetermined operation herein described, a “scissors washing andsterilization operation” among maintenance operations for a medicalinstrument is illustrated as an example.

That is, the control apparatus 10 is in a state where the power isturned on and in a wait state for receiving of a predeterminedinstruction signal is waited for. The wearable terminal apparatus 20 isin a state where the power is turned on, is performing an operation forpicking up a live view image, and is in a state where an instructionsignal is waited for (step S201 illustrated in FIG. 10). Similarly, theauxiliary camera 30 is in a state where the power is turned on, isperforming an operation for picking up a live view image, and is in astate where an instruction signal is waited for (step S301 illustratedin FIG. 11).

In this state, the operator 100 starts the predetermined “scissorswashing and sterilization operation” in a predetermined operation place.First, the operator 100 performs an action for holding an instrument(the scissors 200) to be maintained over a front surface (within animage pickup range) of the auxiliary camera 30 with the instrument inhand in the “scissors washing and sterilization operation”.

Consequently, the wearable terminal apparatus 20 detects whether theaction (the action for holding the scissors 200 over the front surfaceof the auxiliary camera 30; hereinafter abbreviated as “the action forholding the scissors 200 over”) has been performed (step S202illustrated in FIG. 10). The detection processing is performed in theimage judgment unit 21 b in the control unit 21 in the wearable terminalapparatus 20, for example. If “the action for holding the instrumentover” has been detected, the processing proceeds to step S203illustrated in FIG. 10. If the action for holding the instrument overhas not been detected, the wearable terminal apparatus 20 continues torepeat the detection processing.

In step S203 illustrated in FIG. 10, the control unit 21 in the wearableterminal apparatus 20 transmits information indicating that the actionfor holding the instrument over has been performed (hereinafterabbreviated as “information indicating that the instrument has been heldover”) to the control apparatus 10. Then, the control unit 21 enters await state while performing an operation for picking up a live viewimage in step S204.

The control apparatus 10, which is in the wait state confirms whether“the information indicating that the instrument has been held over” hasbeen received in step S101 illustrated in FIG. 9. If the receiving of“the information indicating that the instrument has been held over” hasbeen confirmed, the processing proceeds to subsequent step S102illustrated in FIG. 9. Note that, if the receiving of the informationhas not been confirmed, the control apparatus 10 continues to wait forreceiving of an instruction signal.

In step S102 illustrated in FIG. 9, the control unit 11 in the controlapparatus 10 transmits to the auxiliary camera 30 a request/instruction(an image pickup request instruction) to perform a predetermined imagepickup operation. Then, the control unit 11 waits for an image pickupresult (e.g., image data of the scissors 200) to be transmitted from theauxiliary camera 30.

The auxiliary camera 30 confirms whether the image pickup requestinstruction from the control apparatus 10 has been received in step S302illustrated in FIG. 11. If the receiving of the image pickup requestinstruction has been confirmed, the auxiliary camera 30 performs thepredetermined image pickup operation. Then, the processing proceeds tostep S303 illustrated in FIG. 11.

Image data acquired by the image pickup operation herein performed isimage data of an ambient environment (e.g., an operation room or anoperation table) including the operator 100, for example, in addition toenlarged image data of the scissors 200, for example. It can beconfirmed whether the operation is being performed under a correct(defined) operation environment or whether the operator 100 is wearing acorrect (defined) brace, specifically, a mask, gloves, or a faceprotector, for example, by acquiring the ambient environment data.

In step S303 illustrated in FIG. 11, the auxiliary camera 30 transmitsinformation about image data (mainly still image data) acquired as aresult of the image pickup operation toward the control apparatus 10.Then, the processing returns to step S301. In step S301, the auxiliarycamera 30 waits for the receiving of the predetermined instructionsignal while continuing to perform the operation for picking up a liveview image.

The control apparatus 10 receives the image pickup result by theauxiliary camera 30 (the image data of the scissors 200) (step S102illustrated in FIG. 9).

Note that, if the receiving of the image pickup request instruction hasnot been confirmed in step S302, the processing returns to step S301illustrated in FIG. 11. In step S301 illustrated in FIG. 11, theauxiliary camera 30 waits for the receiving of the predeterminedinstruction signal while continuing to perform the operation for pickingup a live view image.

Returning to FIG. 9 again, if the control apparatus 10 has received theimage pickup result from the auxiliary camera 30 in step S102, theprocessing proceeds to subsequent step S103.

In step S103 illustrated in FIG. 9, the control apparatus 10 controlsthe object judgment unit 11 d, the size judgment unit 11 e, and the likein the control unit 11, to perform processing for judging a maintenancetarget instrument. The judgment processing is processing performed byreferring to the image pickup result (the image data of the scissors200) received in step S102, image data of the maintenance targetinstrument stored in the database 13, and the like.

Then, in step S104 illustrated in FIG. 9, the control apparatus 10similarly controls the object judgment unit 11 d, the size judgment unit11 e, and the like in the control unit 11 to perform processing forconfirming a maintenance environment. This confirmation processing isprocessing performed by referring to the image pickup result (the imagedata of the ambient environment and the operator) received in step S102,corresponding data stored in the database 13, and the like. Then, thecontrol apparatus 10 enters the state where the receiving of thepredetermined instruction signal is waited for, and the processingproceeds to step S105.

In step S105 illustrated in FIG. 9, the control apparatus 10 thenconfirms whether “the information indicating that the tool has been heldover” has been received. If the receiving of “the information indicatingthat the tool has been held over” has been confirmed, the processingproceeds to subsequent step S111 illustrated in FIG. 9. Note that, ifthe receiving of the information has not been confirmed, the controlapparatus 10 continues to wait for the receiving of the instructionsignal (loop processing in step S105).

The operator 100 starts the maintenance tool confirmation operationserving as the “first operation” in the “first process”. As describedabove, an operation performed by the operator 100 in the “firstoperation: tool confirmation” operation is an action for the operator100 holding an appropriate tool (the washing brush 300) used for theoperation over the front surface (within the image pickup range) of theauxiliary camera 30 with the tool in hand.

Consequently, the wearable terminal apparatus 20, which has been in thewait state in the above-described processing in step S204, detectswhether the action (the action for holding the washing brush 300 overthe front surface of the auxiliary camera 30; abbreviated as “the actionfor holding the washing brush 300 over”) has been performed in step S205illustrated in FIG. 10. The detection processing is performed in theimage judgment unit 21 b in the control unit 21 in the wearable terminalapparatus 20, for example. If “the action for holding the tool over” hasbeen detected, the processing proceeds to step S206 illustrated in FIG.10. If “the action for holding the tool over” has not been detected, thewearable terminal apparatus 20 continues to repeat the detectionprocessing.

In step S206 illustrated in FIG. 10, the control unit 21 in the wearableterminal apparatus 20 transmits information about “the action forholding the tool over” (abbreviated as “information indicating that thetool has been held over”) to the control apparatus 10.

The control apparatus 10 in the wait state confirms whether “theinformation indicating that the tool has been held over” has beenreceived in step S105 illustrated in FIG. 9. If the receiving of “theinformation indicating that the tool has been held over” has beenconfirmed, the processing proceeds to subsequent step S111 illustratedin FIG. 9. Note that, if the receiving of the information has not beenconfirmed, the control apparatus 10 continues to wait for the receivingof the instruction signal.

In step S111 illustrated in FIG. 9, the control unit 11 in the controlapparatus 10 controls the operation judgment unit 11 a or the like toperform processing for the maintenance tool confirmation operation (“thefirst operation”).

Then, in step S112 illustrated in FIG. 9, the control apparatus 10transmits an operation confirmation request/instruction to the wearableterminal apparatus 20. The operation confirmation request/instruction isan instruction signal for requesting the movie data acquired by thewearable terminal apparatus 20 after “the information indicating thatthe tool has been held over” has been transmitted.

In step S211 illustrated in FIG. 10, the wearable terminal apparatus 20confirms whether the operation confirmation request/instruction has beenreceived. If the operation confirmation request/instruction has beenreceived, the processing proceeds to subsequent step S212.

Then, in the processing in step S212 illustrated in FIG. 10, thewearable terminal apparatus 20 transmits the acquired movie data to thecontrol apparatus 10. Then, the processing proceeds to step S221.

In step S113 illustrated in FIG. 9, the control apparatus 10 receivesthe movie data transmitted from the wearable terminal apparatus 20 inthe above-described processing in step S211. The control apparatus 10compares the received movie data with the corresponding data stored inthe database 13. The control apparatus 10 confirms whether the operationperformed by the operator 100 is a correct (defined) operation based onthe acquired movie data.

If there is no problem in a result of the above-described comparison,i.e., if it is confirmed that the operation performed by the operator100 (operation recorded in the acquired movie) is a correct (defined)operation (if it is OK) in subsequent step S114, the processing proceedsto subsequent step S116. If it is not OK, the processing proceeds tostep S115.

In step S115 illustrated in FIG. 9, the guide unit 11 c in the controlunit 11 in the control apparatus 10 transmits predetermined informationabout predetermined warning display, guide display, or the like to thewearable terminal apparatus 20. Then, the processing proceeds to stepS116.

In step S221 illustrated in FIG. 10, the wearable terminal apparatus 20confirms whether the information about warning display, guide display,or the like has been received. If the receiving of the information hasbeen confirmed, the processing proceeds to subsequent step S222. If thereceiving of the information has not been confirmed, the processingreturns to step S201. In step S201, the wearable terminal apparatus 20enters the wait state while performing the operation for picking up alive view image.

In step S222, the control unit 21 in the wearable terminal apparatus 20controls the display unit 25, the voice output unit 28, and the like, toperform the warning display, guide display, and the like in apredetermined form.

Examples of the warning display, the guide display, and the like includevoice information display using the voice output unit 28 in addition tovisual information display using the display unit 25. Then, theprocessing returns to the above-described processing in step S201. Instep S201, the wearable terminal apparatus 20 enters the wait statewhile performing the operation for picking up a live view image.

On the other hand, the control apparatus 10 confirms whether the processhas ended in step S116. The confirmation is performed by referring tothe database 13, for example. If it is confirmed that the process hasended, the processing proceeds to subsequent step S117. If it is notconfirmed that the process has ended, i.e., if the process has not endedbut a subsequent operation exists, the processing returns to step S113.

In step S117, the control apparatus 10 performs evidencing processingfor the process which has ended. The evidencing processing is processingfor recording the acquired image data or the like in a predeterminedform, for example. Then, the processing proceeds to step S121.

In step S121, the control apparatus 10 then confirms whether all theprocesses in the operation have ended. If it is confirmed that all theprocesses have ended, the processing proceeds to subsequent step S122.If it is not confirmed that all the processes have ended, the processingproceeds to step S123.

If all the processes in the operation have not ended, the operator 100shifts (changes) a maintenance tool in hand (a tool used in a previousoperation) to a maintenance tool used in a subsequent operation (e.g., asecond operation).

In this case, the control apparatus 10 confirms whether the maintenancetool has been changed in step S123 illustrated in FIG. 9. The processingthen returns to step S105. In step S105 and subsequent steps, thecontrol apparatus 10 repeats similar processing.

On the other hand, when the control apparatus 10 confirms that all theprocesses have ended in the above-described processing in step S121, andthe processing proceeds to step S122, the control apparatus 10 performsfinal evidencing processing in step S122. Examples of the finalevidencing processing include processing for filing the acquiredplurality of image data or the like as a predetermined position filewhile creating a check list or the like based on an OK signal outputtedin the above-described processing in step S114. Then, all the processesend, and the control apparatus 10 returns to the wait state.

As described above, according to the above-described embodiment, theoperation support system 1 can provide operation support for reliablyperforming a predetermined series of operations including a plurality ofoperations according to a defined operation procedure and operationmethod by the two image pickup apparatuses (20, 30) and the controlapparatus 10 acting on each other in cooperation with each other,thereby enabling operation errors to be suppressed.

The operation support system 1 can respectively record as image data andevidence a predetermined action and a used tool during an operation anda situation of an ambient environment or the like. Therefore, theoperation support system 1 can ensure that a predetermined operation hasbeen reliably performed as defined after the operation.

The operation support system 1 can provide support to perform a reliableoperation by effectively utilizing image data acquired during anoperation and eliminating operation errors based on the acquiredplurality of image data and operation information stored in a databasein advance while warning an operator and guiding a correct procedurewhen an operation error has occurred.

The operation errors (e.g., omission of a procedure, a sequential errorin a procedure, and negligence in a defined operation) can be reliablyeliminated. According, everyone can smoothly and reliably perform thedefined operation while evidence recording of a passage of the operationcan also be easily performed.

Note that, while all the first process to the fourth process areperformed in the same place in the example of the operation (theinstrument washing and sterilization operation) described in theabove-described embodiment, the present invention is not limited to sucha form. For example, all the plurality of operation processes may not beperformed in the same place depending on the operation but the place maybe changed when some of the operation processes are executed.

For example, in the operation example (the instrument washing andsterilization operation) described in the above-described embodiment,the first process and the second process may be performed on theoperation table 150 including the washing tank 150 a, and the thirdprocess and the fourth process may be performed using another operationtable (not illustrated) in a different place.

The reason is that the first process and the second process are each awashing operation performed in a plumbing facility, while the thirdprocess and the fourth process are each desirably an operation performedin a dry place. In such a case, an auxiliary camera serving as anotherindividual equivalent to the above-described auxiliary camera 30 may bedisposed on the other operation table (not illustrated) on which theoperations in the third process and the fourth process are performed,and the other auxiliary camera may be configured to act under thecontrol of the above-described control apparatus 10 in cooperation withthe control apparatus 10.

Although an example in which the auxiliary camera 30 is placed on theoperation table 150 has been described in the above-describedembodiment, the arrangement of the auxiliary camera 30 is not limited tothe arrangement in the example.

For example, the auxiliary camera 30 can also be disposed in a liquid inthe washing tank 150 a. FIG. 16 illustrates how the auxiliary camera 30thus configured is. FIG. 16 is a conceptual diagram illustrating how apredetermined maintenance operation (instrument washing andsterilization operation) is being performed using an operation supportsystem according to a modification to the above-described embodiment.FIG. 17 is a diagram illustrating one scene of movie data acquired by awearable terminal apparatus in the modification illustrated in FIG. 16.FIG. 18 is a diagram illustrating still image data acquired by anauxiliary camera in the modification illustrated in FIG. 16.

In this modification, an auxiliary camera 30A disposed in a liquid is animage pickup apparatus having a waterproof function. If the auxiliarycamera 30A is thus disposed in the liquid in the washing tank 150 a, anoperation in the liquid can be more clearly recorded.

Note that in the configuration, communication between the auxiliarycamera 30A and the control apparatus 10 needs to be performed using awired cable or exposing a wireless communication antenna to a watersurface.

As a form of the wearable terminal apparatus 20, a wearable terminalapparatus can also have an image pickup function integrally incorporatedinto a face protector or the like to be always worn by an operator whenan operation is performed, for example, in addition to the formsillustrated in the above-described embodiment and modification. In thewearable terminal apparatus thus configured, the operator needs toalways wear the face protector before the operation. Thus, a burden ofwearing a separate wearable terminal apparatus can be removed, andsimultaneously an operator's error of forgetting to wear the wearableterminal apparatus can be suppressed.

If a control unit is put into the wearable terminal apparatus, thewearable terminal apparatus may be changed to a wearable apparatusincluding not only an image pickup unit configured to acquire eye imagedata to acquire operation information corresponding to the eye line ofthe operator but also a communication unit having a database storingoperation information composed of image information from a plurality ofvisual fields relating to a predetermined operation in advance andconfigured to acquire an auxiliary image from an auxiliary cameraapparatus provided at a different position from a position of theoperator and an operation judgment unit configured to search for andjudge the operation information in the database according to first imagedata and second image data, or may be changed to a system including adatabase outside the wearable apparatus and configured to search thedatabase. The same applies to the auxiliary camera.

Note that a portion described as a “section or unit” in the embodimentmay be configured by a dedicated circuit or a combination of a pluralityof general-purpose circuits, or may be configured by being combined witha microcomputer configured to operate according to preprogrammedsoftware, a processor such as a CPU, or a sequencer such as an FPGA(field programmable gate array), as needed. Such a design that anexternal apparatus assumes a part or the whole of control of the unit ispermitted. In this case, a wired or wireless communication circuit isinterposed between the section or unit and the external apparatus.Communication may be performed by Bluetooth, WiFi (Wireless Fidelity), atelephone line, or the like, or may also be performed by a USB(universal serial bus). The dedicated circuit, the general-purposecircuit, and a control unit may be integrally configured as an ASIC(application specific integrated circuit). A movement unit, for example,includes various types of actuators and includes a connection mechanismfor movement, as needed, and the actuators operate by a driver circuit.The driver circuit is also controlled by a microcomputer, an ASIC, orthe like according to a specific program. The control may bespecifically corrected or adjusted, for example, according toinformation outputted by various types of sensors and peripheralcircuits of the sensors.

Each of processing sequences described in the one above-describedembodiment can permit a change of the processing sequence as long as thechange is not contrary to the principle. Therefore, for theabove-described processing sequence, an execution order of processingsteps may be changed, the plurality of processing steps may besimultaneously executed, or the order of processing steps may differ foreach execution of the series of processing sequences, for example. Thatis, even if operational flows are described using “first”, “next”, andthe like for convenience in the claims, specification, and drawings, thedescription does not necessarily mean that operations are to beperformed in this order. In each of the steps constituting theoperational flows, it is needless to say that a portion, which does notaffect the essence of the invention, can also be omitted, as needed.

In a technology herein described, most of control and functions mainlydescribed in the flowcharts can be set by a software program, and thecontrol and functions can be implemented by a computer reading andexecuting the software program.

Although description has been made by providing a control unitconfigured to control all devices in cooperation with one another forsimplicity of illustration, each of the devices may be made to have acooperation control function. A wearable apparatus may have such acontrol function, or a part of a control apparatus can also beconsidered as a wearable apparatus. An “auxiliary camera” may have afunction of the control apparatus to control the wearable apparatus, ora system in which the control apparatus includes the auxiliary cameramay be used. Further, some apparatuses may function as the controlapparatus as a whole by transmitting a progress of each of steps to oneanother by communication.

The software program is electronic data the whole or a part of which isstored or recorded in advance in the above-described storage medium orstorage unit, specifically, a storage medium such as a portable mediumsuch as a nonvolatile memory such as a flexible disk CD-ROM (compactdisc read only memory) or a volatile memory such as a hard disk inproduct manufacturing processes as a computer program product. Thestorage media may be arranged in a distributed manner for the devices toshare control. Alternatively, the software program can be distributed orprovided at the time of product shipping or via a portable medium or acommunication line. A user can operate the software program via acommunication network, the Internet, or the like by downloading thesoftware program and installing the downloaded software programs into acomputer or installing those software programs into the computer fromthe storage media, and thus can easily implement an image pickupapparatus according to the present embodiment.

The present invention is not limited to the above-described embodiment,and it is needless to say that various modifications and applicationscan be implemented without departing from the spirit of the invention.Further, the above-described embodiment includes inventions in variousstages, and the various inventions can be extracted by an appropriatecombination of a plurality of constitutional elements disclosed. Even ifsome of all the constitutional elements described in the above-describedembodiment are deleted, for example, if the issue to be solved by theinvention can be solved and the effect of the invention can be obtained,a configuration from which the constitutional elements are deleted canbe extracted as the invention. Further, the components over differentembodiments may be combined, as needed. The invention is not restrictedby specific embodiments except as by the appended claims.

What is claimed is:
 1. An operation support system comprising: a firstimage pickup device configured to acquire movie data associated with anoperation; a second image pickup device configured to acquire stillimage data associated with the operation; a control circuit incommunication with the first image pickup device, the second imagepickup device, and a database, the control circuit configured to controlthe first image pickup device and the second image pickup device incooperation with each other while acquiring predetermined operationinformation from the movie data at a time when the operation isperformed and ambient information on an operation target or operationtool from the still image data; and the database is configured to storeoperation information corresponding to a predetermined operation and toassociate the predetermined operation information obtained from themovie data with the ambient information obtained from the still imagedata; wherein the control circuit compares the predetermined operationinformation obtained from the movie data and an image of an operationtarget or operation tool obtained from the still image data respectivelywith the operation information in the database, and judges whether theoperation is correctly performed.
 2. The operation support systemapparatus according to claim 1, wherein the first image pickup device isa wearable terminal apparatus configured to acquire movie data in animage pickup range substantially equivalent to a field of view of anoperator and worn by the operator, the second image pickup device is anauxiliary camera configured to acquire still image data in apredetermined image pickup range including a predetermined object from adifferent direction from a direction of the wearable terminal apparatus,and the control circuit controls the auxiliary camera to acquire stillimage data according to the movie data acquired by the wearable terminalapparatus.
 3. The operation support system according to claim 1, whereinthe first image pickup device further includes a display, and thecontrol circuit displays a warning using the display when the controlcircuit judges an operation error.
 4. The operation support systemaccording to claim 3, wherein the display is a display panel configuredto display visual information.
 5. The operation support system accordingto claim 3, wherein the display is a sound production device configuredto sound voice information.
 6. The operation support system according toclaim 1, wherein the control circuit further includes an evidencerecorder, and the evidence recorder records the movie data acquiredduring the operation by the first image pickup device and the stillimage data acquired during the operation by the still image data.
 7. Awearable apparatus comprising: a first image pickup device configured toacquire first image data during an operation with operation informationcorresponding to the direction of an eye line of an operator; acommunication unit configured to: acquire second image data from asecond image pickup device with ambient information relating to theoperation, the second image pickup device provided at a differentposition from a position of the operator; and communicate with adatabase configured to store operation information corresponding to apredetermined operation and to associate the first image data with thesecond image data; and an operation judgment circuit configured to judgewhether the operation is correctly performed by comparing the firstimage data and the second image data with the operation informationstored in the database.
 8. An image pickup apparatus comprising: areceiver configured to: receive first information from a wearable imagepickup device configured to acquire first image data during an operationwith operation information corresponding to a direction of an eye lineof an operator during an operation; receive second information from anauxiliary image pickup device configured to acquire second image datawith ambient information relating to the operation, the auxiliary imagepickup device provided at a different position from the direction of theeye line of the operator; and a database storing defined operationinformation about a predetermined operation in advance and to associatethe first information with the second image information; and anoperation judgment circuit configured to compare the first informationand the second information with the defined operation information in thedatabase to determine whether the predetermined operation was correctlyperformed.
 9. An operation support method comprising: acquiring firstimage data during an operation with operation information correspondingto the direction of an eye line of an operator; acquiring second imagedata serving as operation information with ambient information relatingto the operation acquired at a different position from the first imagedata; communicating the acquired first image data and second image datato a control circuit in communication with a database, the databasestoring defined operation information including image information from aplurality of visual fields relating to a predetermined operation inadvance; and judging whether the operation is correctly performed bycomparing the first image data and the second image data with thedefined operation information stored in the database.
 10. An operationsupport system comprising: a wearable terminal device configured toacquire movie data associated with an operation in an image pickup rangesubstantially equivalent to a field of view of an operator and worn bythe operator; an auxiliary camera configured to acquire still image dataassociated with the operation in a predetermined image pickup rangeincluding a predetermined object from a different direction from adirection of the wearable terminal device; a control circuit incommunication with the wearable terminal device, the auxiliary camera,and a database, the control circuit configured to control the wearableterminal device and the auxiliary camera in cooperation with each otherwhile acquiring predetermined operation information at a time when theoperation is performed from the movie data and information on an ambientenvironment from the still image data; and the database is configured tostore operation information corresponding to a predetermined operationand to associate the predetermined operation information obtained fromthe movie data with the information on the ambient environment obtainedfrom the still image data, wherein the auxiliary camera is controlled,based on the movie data acquired by the wearable terminal device, toacquire still image data serving as evidence of an operation, and thecontrol circuit compares the predetermined operation informationobtained from the movie data and the information on the ambientenvironment obtained from the still image data with the operationinformation stored in the database, and judges whether the operation iscorrectly performed.
 11. An operation support method comprising:acquiring movie data associated with an operation in an image pickuprange substantially equivalent to a field of view of an operator;acquiring still image data associated with the operation in apredetermined image pickup range including a predetermined objectaccording to the acquired movie data from a different direction from adirection of the movie data as evidence of an operation; communicatingthe acquired movie data and the acquired image data to a control circuitin communication with a database storing defined operation information;associating the movie data with the still image data; and judgingwhether the predetermined operation was correctly performed by comparingthe movie data and the image data with the defined operation informationstored in the database.